Crowd-sourced data coding for the social sciences: massive non-expert human coding of political texts

A large part of empirical social science relies heavily on data that are not observed in the field, but are generated by researchers sitting at their desks. Clearly, third party users of such coded data must satisfy themselves in relation to both reliability and validity. This paper discusses some of these matters for a widely used type of coded data, derived from content analysis of political texts. Comparing multiple “expert” and crowd-sourced codings of the same texts, as well as with independent estimates of the same latent quantities, we assess the extent to which we can estimate these quantities reliably using the cheap and scalable method of crowd sourcing. Our results show that, contrary to naive preconceptions and reflecting concerns that are often swept under the carpet, a set of expert coders is also a crowd. We find that deploying a crowd of non-expert coders on the same texts raises issues relating to coding quality that need careful consideration. If these issues can be resolved by careful specification and design, crowdsourcing offers the prospect of cheap, scalable and replicable text coding. While these results concern text coding, we see no reason why they do not extend to other forms of expert coded data in the social sciences. Paper prepared for presentation at the 70 Annual Conference of the Midwest Political Science Association. Palmer House Hotel, Chicago. 12-15 April 2012. We thank Joseph Childress at CrowdFlower for assisting with the setup of the crowd-sourcing platform. This research was supported financially by the European Research Council grant ERC-2011-StG 283794-QUANTESS. Crowd-sourced data coding for the social sciences / 1